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INTRODUCTION 


Telepresence  and  the  Effect  of  Time  Delay 

The  fundamental  purpose  of  a  telepresence  system  is  to  extend  an 
operator's  sensory-motor  facilities  and  problem  solving  abilities  to  a  remote 
environment  [21].  Telepresence  is  achieved  by  projecting  an  operator’s 

manipulatory  dexterity  to  a  remote  environment  while  reflecting  sensory 
information  from  that  environment  so  realistically  that  the  operator  feels 

present  at  the  remote  site  [1].  The  illusion  of  presence  is  dependent  upon  the 
fidelity  and  scope  of  the  interface.  A  fundamental  requirement  of  presence  is 

a  strong  correlation  between  the  operator's  movements,  as  perceived 

proprioceptively  and  kinesthetically  by  the  operator,  and  the  corresponding 
manipulations  in  the  remote  site,  as  perceived  visually  and  haptically  by  the 
operator  [11].  Time  delay  between  master  and  slave  distorts  the  correlation 
between  kinesthetic  sense  and  feedback  from  the  remote  environment, 
destroys  the  illusion  of  presence,  and  ultimately  degrades  performance  within 
the  remote  environment  [5].  Time  delays  as  short  as  100  ms  have  been  found  to 
degrade  performance  in  telemanipulation  tasks.  Because  destructive  time 
delays  are  inherent  to  all  telepresence  systems  in  which  the  remote  sites 
require  round  trip  communication  transmission  time  of  more  than  100  ms, 
much  effort  has  gone  into  studying  and  eliminating  the  performance 
degradation  resulting  from  such  delays. 

Ferrel  (1965)  was  the  first  to  demonstrate  experimentally  the 
performance  degradation  resulting  from  time  delays  in  teleoperation  [6,  7]. 
He  found  that  operators  confronted  with  time  delay  had  a  tendency  to  adopt  a 
move-and-wait  strategy.  Thompson  (1977)  showed  how  task  completion  times 
were  slowed  by  both  time  delay  and  reduction  in  degrees  of  constraint  in  part 
mating  tasks  [33].  Since  that  time,  much  research  has  been  devoted  toward 
combating  the  effect  of  temporal  delays  through  supervisory  control  and 
predictive  displays.  Supervisory  control  solves  the  time  delay  problem  by 
simply  removing  the  human  from  the  control  loop.  Supervisory  systems  allow 
an  operator  to  specify  high-level  commands  to  an  autonomous  manipulator  in 
a  remote  environment  [4].  Although  this  technique  is  valuable  for  tasks  that 
can  be  performed  by  an  autonomous  manipulator,  it  does  not  solve  the  time 
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delay  problem  for  those  cases  when  human  presence  is  required.  Predictive 
displays,  on  the  other  hand,  combat  the  time  delay  problem  while  still 
maintaining  some  degree  of  operator  presence  in  the  remote  environment. 
Rather  than  providing  the  operator  with  lime  delayed  feedback  from  a  remote 
worksite,  a  predictive  display  provides  the  operator  with  simulated  sensory 
feedback  generated  through  simulated  interaction  with  a  computer  model  of 
the  remote  environment.  The  interaction  with  the  computer  model  can  occur 
m  real  time,  while  the  real  telemanipulation  task  actually  occurs  some  time 
delay  later.  In  theory,  interaction  with  a  highly  accurate  and  perceptually 
rich  model  of  a  remote  environment  v/ould  provide  an  operator  with 
equivalent  sensory  feedback  to  actual  telepresence  in  the  real  remote 


environmen 
displays  to 


In  practice,  computational  requirements  limit  most  predictive 


.ghly  simplified  and  abstracted  visual  feedback 


from  the  simulated 


^mote  site 


few  cases,  limited  use  of  other  sensory  modalities 


miormatu 


incorporate 


}redictive 


An  aiternative  or  supplement  to  the  implementation  of  either 
supervisory  control  or  predictive  display  methods  is  the  use  of  virtual  fixtures 
to  reduce  performance  degradation  due  to  time  delay  in  remote  manipulation 
tasks.  Because  the  use  of  virtual  fixtures  does  not  involve  modeling  of  the 
workspace,  detailed  previous  knowledge  of  the  remote  site  is  not  required. 
Thus,  virtual  fixturing  is  particularly  useful  for  enhancing  lime  delayed 
teleoperation  tasks  in  unstructured,  changing  environments  where  other 
methods  fail.  Before  describing  an  empirical  study  which  demonstrates  the 
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effectiveness  of  virtual  fixtures  to  reduce  the  performance  degradation  due  to 
time  delays,  a  brief  review  of  the  virtual  fixture  concept  is  given. 

The  Concept  of  Perceptual  Overlays 

When  asked  to  draw  a  straight  line  in  the  real  world,  human 
performance  can  be  greatly  enhanced  by  using  a  simple  tool  such  as  a  ruler. 

The  use  of  a  ruler  reduces  the  amount  of  mental  processing  required  to 
perform  the  task,  speeds  an  operator's  line  drawing  ability,  and  most  of  all 

allows  an  operator  to  draw  a  significantly  better  line  than  if  no  ruler  had  been 
used  at  all.  Without  a  ruler,  line  drawing  is  a  manual  task  that  requires 
constant  visual  supervision  and  hand/eye  coordination.  With  a  ruler,  line 
drawing  is  not  only  faster  and  straighter,  but  the  dependence  on  visual 
feedback  is  reduced,  freeing  up  that  modality  for  other  uses.  What  is  more,  a 
ruler  is  often  used  as  a  barrier  to  protect  against  dangerous  or  destructive 
failures,  protecting  the  work-piece  from  the  slip  of  a  pencil  or  knife.  Such 
guidance  and  protection  allows  the  operator  to  ease  mental  criteria  for  task 

success  and  failure,  reducing  the  level  of  concentration  devoted  to  the  task. 

Although  a  simple  tool  by  any  standard,  a  common  ruler  is  clearly  a  powerful 
performance  aid  in  manual  line  drawing  tasks. 

Although  the  use  of  a  ruler,  or  any  straight-edge,  to  assist  in  straight- 

line  drawing  is  an  effective  means  of  enhancing  human  performance  in  a 
manual  task,  can  such  a  process  be  generalized  beyond  line  drawing?  Ruler 
use  can  be  thought  of  as  nothing  more  than  a  process  of  overlaying  abstract 

sensory  information  on  top  of  a  workspace.  Thus,  a  ruler  can  be  generalized 

as  a  particular  “perceptual  overlay”  designed  to  enhance  operator 

performance  in  line  drawing  tasks  [26].  In  the  particular  case  of  a  ruler,  the 
overlaid  sensory  information  represents  a  single  rigid  surface  to  be  perceived 

haptically  and  visually  by  the  user.  By  overlaying  this  additional  sensory 

information  on  top  of  the  workspace,  the  mental  and  physical  demands  of  the 
straight-line  drawing  task  are  reduced  and  operator  performance  is  enhanced. 

If  a  simple  ruler-like  perceptual*  overlay  can  so  greatly  enhance  the 

performance  of  a  real  world  manipulatory  task  such  as  straight-line  drawing, 
it  seems  that  computer  generated  perceptual  overlays  could  be  developed 
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within  virtual  environments  to  enhance  the  performance  of  telemanipulation 
tasks  within  remote  worksites.  Just  as  a  ruler  can  be  overlaid  on  top  of  a  real 
workspace,  such  virtual  perceptual  overlays  could  be  overlaid  on  top  of  the 

sensory  feedback  from  a  remote  workspace. 

The  Virtual  Fixture  Metaphor 

Because  the  abstract  notion  of  overlaid  sensory  information  is  as 

difficult  to  conceptualize  as  it  is  to  talk  about,  the  virtual  fixture  metaphor  was 
introduced  as  a  means  of  describing  such  computer  generated  sensations  as 

concrete  physical  structures  [Rosenberg  1992],  It  must  be  stressed  that  the 
point  of  this  metaphor  is  intended  to  facilitate  the  understanding  of,  and 

interaction  with,  perceptual  overlays  and  should  not  be  taken  so  literally  as  to 
limit  the  scope  of  the  perceptual  overlay  concept.  Virtual  fixtures  are  thus 
defined  as  abstract  sensory  information  overlaid  on  top  of  reflected  sensory 
feedback  from  a  remote  environment.  Although  overlaid  on  top  of  the  user’s 

perception  of  the  remote  environment,  virtual  fixtures  are  completely 
independent  of  all  information  from  the  remote  site  and  are  thus  immune 

from  communication  delays  and  bandwidth  limitations. 

Like  the  ruler  guiding  the  pencil,  virtual  fixtures  overlaid  on  top  of  a 
remote  workspace  could  act  to  reduce  mental  processing  required  to  perform 

the  task,  reduce  the  work  load  of  certain  sensory  modalities,  and  most  of  all 
allow  precision  and  performance  to  exceed  natural  human  abilities.  Although 
virtual  fixtures  could  be  functionally  equivalent  to  fixtures  in  the  real  world, 
there  are  many  advantages  inherent  to  virtual  fixtures  because  they  are 

computer  simulations  rather  than  real  physical  hardware.  When  overlaid  on 

top  of  a  remote  workspace,  the  fixtures  only  interact  with  the  user  and  not 

with  the  workspace  itself.  Thus,  virtual  fixtures  can  occupy  the  same  physical 

space  as  real  objects  in  the  remote  workspace.  This  means  that  the  workspace 

geometry  imposes  no  constraints  upon  the  placement  or  configuration  of 
virtual  fixtures.  What  is  more,  such  fixturing  has  no  mass,  no  physical  or 
mechanical  constraints,  requires  no  machining  time  or  maintenance,  can  be 
easily  prototyped  and  modified,  and  can  essentially  be  transported  to  remote 
locations  using  nothing  more  than  standard  communication  links. 
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If  we  explore  the  concept  of  virtual  fixtures  using  the  simple  ruler 
example  as  the  starting  point,  the  first  elements  to  consider  might  be  rigid 
planar  surfaces.  Such  fixtures  would  be  composed  of  haptic  sensations 
generated  by  reflecting  simulated  forces  to  the  operator  through  a  force- 
reflecting  master.  As  the  operator  interacts  with  the  modeled  surfaces,  the 
reaction  forces  would  be  computed  and  reflected  appropriately.  Of  course, 
such  fixtures  are  by  no  means  limited  to  rigid  surfaces.  Abstracting  the 
fixturing  concept,  we  might  consider  modeling  compliant  surfaces,  damped 
surfaces,  even  viscous  or  coulomb  frictional  contacts.  In  fact,  the  simulation 
environment  offers  such  freedom  that  virtual  fixtures  could  even  be 
developed  as  attractive  or  repulsive  fields.  Although  fixtures  composed  of 
haptic  sensations  offer  endless  possibilities,  the  fixturing  concept  is  not 
limited  to  that  modality.  Abstract  fixtures  could  be  composed  of  visual, 
auditory,  even  tactile  sensations  used  alone  or  in  cross-modal  combinations. 
For  example,  if  a  haptic  fixture  composed  of  rigid  planar  surfaces  or  attractive 
force  fields  was  developed  to  aid  a  particular  task,  an  audio,  video,  or  vibratory 
signal  could  be  mapped  to  various  locations  along  the  fixture  to  enhance 
interaction.  Such  additional  modalities  could  be  used  to  indicate  deviations 
from  a  trajectory,  proximity  to  a  danger  zone,  even  provide  feedback  of 
velocities  or  accelerations. 

Abstracting  the  fixturing  concept  further,  we  could  imagine  virtual 
fixtures  imbued  with  particular  visual  qualities  to  enhance  interaction  with 
environments.  For  example,  virtual  fixtures  composed  of  haptic  surfaces  could 
be  modeled  with  optical  properties  to  suit  the  task  at  hand.  The  fixture  might 
be  invisible  to  the  user  if  the  operator  gains  no  benefit  from  visual  cues,  it 
could  be  made  to  look  like  a  solid  object  if  rich  visual  cues  are  useful  for  the 
task,  it  could  even  be  made  to  look  like  a  transparent  glassy  solid  if  visual  cues 
are  important  but  the  user  wants  to  avoid  obscuring  the  workspace.  Fixtures 
might  even  be  designed  as  visual  filters  to  block  particular  distractions, 
enhance  contrast,  provide  depth  cues,  even  magnify  a  part  of  the  workspace. 
One  can  even  imagine  the  benefit  of  a  fixture  composed  of  compliant  surfaces 
which  changes  color  or  brightness  with  compression.  Previous  work  with 
targeting  cues  [3],  predictive  displays  [2,  19,  27,  28,  29],  and  perspective 
overlays  [14,  32]  has  demonstrated  that  overlaid  visual  cues  can  enhance 
performance  and  understanding  within  a  teleoperation  environment. 
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If  the  description  of  virtual  fixtures  thus  far  seems  too  abstract,  a  simple 
example  may  drive  the  concept  home.  Imagine  a  situation  where  a 

teleoperating  surgeon  performs  a  delicate  procedure  on  a  patient.  Although 

such  uses  of  telerobotics  are  still  in  the  research  phase,  it  is  an  application 
that  demands  a  high  degree  of  human  performance  within  a  remote 
workspace.  Now  imagine  that  a  virtual  fixture  is  being  used  by  the  doctor  to 
enhance  his  abilities  in  this  procedure.  The  fixture  might  appear  to  the 

doctor  like  a  flat  plane  of  glass  with  a  grooved  guide  for  the  scalpel.  The 

glassdike  virtual  fixture  might  actually  pass  directly  through  a  patient’s 
body,  preventing  the  scalpel  from  penetrating  below  a  particular  depth  but 
not  obscuring  vision  of  the  tissue  below.  By  sliding  the  scalpel  along  the  edge 
of  a  groove  in  the  fixture,  the  surgeon  could  make  a  perfect  incision.  The 

slightest  deviation  from  the  target  trajectory  might  be  reported  by  an  audio  or 
tactile  signal.  The  power  and  flexibiliiy  of  such  a  system  would  be  unmatched 


y  actual  physical  tooling.  Besides  the  fact  that  such  a  fixture  in  the  real 
world  could  net  pass  directly  through  a  patient's  body,  it  could  not  be  put  in 
place  at  the  touch  of  a  button,  removed  at  the  touch  of  another  button,  or 
easily  altered  as  conditions  change.  What  is  more,  virtual  fixturing  does  not 
nave  to  be  fabricated,  sterilized,  monitored,  or  maintained. 


Virtual  Fixtures  and  Time  Delay 


As  demonstrated  in  a  compaiiiGn  study,  the  use  of  simple  virtual  fi^utures 
greatly  enhances  operator  performance  in  a  telemanipuiation  task  without 
'dme  delay  [26].  The  results  showed  that  operator  performance  was  increased 
up  to  70%  when  simple  virtual  fixtures  were  used  as  compared  to  the  no-* 
fixture  case.  The  results  suggested  that  virtual  fixtures  enhance  performance 
by  simplifying  the  conceptualization  of  the  task,  providing  localizing 
references  to  the  remote  worksite,  and  by  reducing  the  demands  on  taxed 
sensory  modalities  by  providing  perceptual  information  to  multiple  sensory 
pathways.  When  time  delays  are  introduced  into  teleoperation  systems, 

information  lag  between  the  user  space  and  the  remote  workspace  degenerates 
performance  by  distorting  the  correlation  betv/een  the  user’s  proprioceptive^ 
kinesthetic  sense  of  self  dynamics  and  the  sensory  feedback  from  the  remote 
worksite.  Such  time  distortions  hinder  user  localization  to  the  remote 
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workspace  and  reduce  the  sensation  of  remote  presence  [5].  Because  of  the 
perceptual  problems  that  time  dela3^s  introduce,  it  is  believed  that  the 

beneficial  effects  previously  shown  to  be  provided  by  virtual  fixtures  in 

teleoperation  without  time  delay  will  be  even  more  pronounced  when  time 

delays  are  included.  This  study  investigates  the  question  of  whether  the 

application  of  simple  virtual  fixtures  can  be  effective  in  reducing 

performance  degradation  due  to  the  introduction  of  time  delay  into 
teleoperation. 

It  should  be  pointed  out  that  although  a  virtual  fixture  is  overlaid  on  top 
of  the  reflection  of  the  remote  environment,  it  is  completely  independent  of 
all  sensory  feedback  from  that  remote  environment.  Because  the  virtual 

fixture  does  not  depend  on  information  being  fed  back  from  the  remote  site, 

user  interaction  with  the  fixture  is  immune  to  the  temporal  distortions  that 

result  from  communication  delays.  Virtual  fixtures  therefore  provide  the 

operator  with  real-time  reference  points  that  are  overlaid  on  lop  of  the  time 

distorted  reflection  of  the  remote  site.  It  is  hypothesized  that  the  introduction 

of  such  real-time  sensory  cues  can  counteract  the  effect  that  time  delay  has  in 

weakening  the  correlation  between  the  operator's  kinesthetic  sense  of  self 
dynamics  and  the  sensory  feedback  of  remote  manipulator  dynamics.  By 

providing  sensory  information  that  can  be  interacted  with  in  real  lime, 

virtual  fixtures  are  expected  to  enhance  user  localization  to  the  remote 

worksite  and  strengthen  the  overall  illusion  of  remote  presence.  Because  a 
weakened  correlation  between  kinesthetic  sense  and  sensory  feedback  is  a 

primary  source  of  performance  degradation  due  to  time  delay,  it  is 
hypothesized  that  the  use  of  virtual  fixtures  will  greatly  enhance  operator 
performance  in  time  delayed  leleoperalion  tasks. 

Consider  a  teleopcrated  line  drawing  task,  as  described  in  a  previous 

example,  but  this  time  include  time  delay  in  the  scenario.  The  teleoperalor  is 
controlling  a  dexterous  remote  manipulator  that  is  holding  a  pencil  in  its 
gripper.  The  operator  is  given  the  simple  task  of  drawing  a  straight  line  on  a 
piece  of  paper  in  the  remote  environment.  Because  drawing  a  straight  line 

requires  constant  visual  supervision  and  hand/eye  coordination,  even  a  small 
delay  between  the  operator's  imparted  hand  motion  and  visual  feedback  from 

the  remote  site  can  greatly  impair  operator  performance.  Such  tasks  often 
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result  in  operator  instability  when  the  user  tries  to  compensate  for  deviations 
from  the  desired  trajectory  with  real-time  manual  adjustments  even  though 
the  user  is  being  provided  with  time  delayed  sensory  feedback  of  the 
trajectory  deviation  errors.  Lines  drawn  under  such  conditions  often  reveal  a 
characteristic  oscillation  resulting  from  the  operator  trying  to  react  to  past 
events. 

Predictive  displays  have  been  implemented  to  combat  these  sorts  of  time 
delay  problems  by  providing  the  operator  with  sensory  feedback  from  a 
computer  simulation  of  the  remote  workspace  that  can  be  interacted  with  in 
real  time.  For  the  above  line  drawing  telemanipulation  task,  the  user  of  a 
predictive  display  would  perform  the  task  by  interacting  with  a  graphic 
computer  model  the  remote  pencil  and  paper  workspace  rather  than  with  the 
actual  time  delayed  feedback  from  the  remote  site.  If  the  model  of  the  remote 
manipulator  holding  the  pencil  is  accurate,  the  user’s  interaction  with  the 
simulation  will  cause  the  real  pencil  to  draw  a  straight  line  in  the  real  remote 
environment  at  some  time  delay  after  the  user  finishes  manipulating  the 
predictive  model.  Although  the  use  of  a  predictive  display  is  a  viable  solution 
to  time  delay  problems  in  many  simple  telemanipulation  tasks,  the 
requirement  for  accurate  modeling  of  the  remote  manipulator  and  remote 
workspace  restricts  the  use  of  predictive  displays  to  structured  environments 
where  the  workspace  and  task  are  well  known  in  advance. 

A  proposed  alternative  or  supplement  to  the  use  of  predictive  displays, 
which  does  not  require  a  detailed  modeling  of  the  remote  workspace,  is  the 
implementation  of  virtual  fixturing.  Rather  than  simulating  a  real-time  model 
of  the  remote  pencil  and  paper  worksite,  the  virtual  fixture  approach  would  be 
to  simply  provide  the  user  with  a  real-time  model  of  a  virtual  ruler  that  can  be 
overlaid  on  top  of  the  reflection  of  the  real  remote  environment.  Such  a  ruler 
would  be  a  generic  virtual  fixture,  chosen  from  a  toolbox  of  fixtures,  that  could 
be  overlaid  on  top  of  an  arbitrary  remote  environment.  The  use  of  a  virtual 
ruler  in  such  a  line  drawing  task  would  likely  shift  the  user’s  sensory 
dependency  from  the  time  delayed  visual  feedback  reflected  from  the  remote 
site  to  the  real-time  haptic  feedback  provided  by  the  fixture.  As  a  result,  the 
performance  degradation  due  to  time  delay  would  likely  be  reduced.  What  is 
more,  the  kinematic  constraints  and  haptic  cues  provided  by  the  ruler-like 
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virtual  fixture  would  likely  result  in  the  user  drawing  a  much  straighter  line 
than  if  no  fixture  had  been  used.  Of  course  this  simple  example  of  virtual 
fixture  implementation  can  be  extended  to  much  more  complex 

multidimensional  tasks  as  well  as  more  abstract  virtual  fixturing.  It  is  in  these 
more  complex  situations  where  the  application  of  virtual  fixturing  is  most 

appropriate. 


Perceptual  Workstation  Environment 

Although  the  notion  of  virtual  fixturing  does  seem  promising,  if  the 
development  of  fixtures  requires  complex  computation  or  intimate  knowledge 

of  the  workspace  to  be  effective,  robot  autonomy  or  predictive  displays  might 
be  a  preferred  solution  to  the  task  at  hand.  If,  on  the  other  hand,  effective 
fixtures  could  be  developed  out  of  basic  building  blocks  and  quickly 
implemented  by  a  teleoperator  in  an  interactive  environment,  fixtures  could 

be  used  in  unstructured  or  changing  environments  unsuitable  for  autonomous 
systems.  Thus  it  is  proposed  that  a  workstation  type  environment  be  developed 
to  allow  leleoperators  to  design  and  implement  assistive  fixtures  when 
confronted  with  an  unknown  task  in  an  unstructured  environment  upon  first 

encounter.  With  such  a  workstation  in  mind,  the  study  described  in  this  paper 

investigates  the  use  of  simple  haptic  surfaces  as  perceptual  aids  in  a  basic  peg 
insertion  task.  Although  a  workstation  environment  could  implement  more 
sophisticated  surfaces  or  fields  including  a  more  diverse  array  of  sensory 
modalities,  it  was  thought  that  if  simple  combinations  of  haptic  surfaces  could 
be  made  into  effective  fixtures  for  time  delayed  telemanipulation,  the 
potential  of  virtual  fixtures  would  be  adequately  displayed. 
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TELEOPERATOR  PERFORMANCE  ASSESSMENT 


Fitts*  Law  Performance  Test 


To  quantify  teleoperator  performance  in  a  remote  manual  task,  a  Fitts’ 
Law  paradigm  was  chosen  because  of  its  general  acceptance  as  a  robust 

measure  of  human  performance.  Although  extensive  use  of  Fitts’  Law  is 
documented  in  human  performance  literature,  little  work  has  been  done  to 
extend  the  paradigm  to  a  telepresence  environment.  McGovern  (1975)  used  a 
Fitts’  Law  task  to  demonstrate  the  merit  of  a  closed  loop  master-slave  system  as 
compared  to  an  open  loop  exoskeletal  system.  Hill  (1979)  used  a  Fitts’  task  to 
demonstrate  performance  differences  as  a  function  of  force  feedback  from 
the  manipulative  system.  Pepper  (1988)  was  the  first  to  use  the  Fitts’  task  in  a 
true  telepresence  scenario.  While  previous  work  had  employed  Fitts’  Law  with 
the  teleoperator  in  direct  view  of  the  workspace,  this  work  used  Fitts’  Law  to 
compare  a  variety  of  viewing  conditions  which  included  remote  visual  links 
oelween  operator  and  workspace.  These  studies  have  shown  that  a  Fitts’  Law 
paradigm  is  appropriate  for  analysis  of  perceptual-motor  performance  v/ithin 
teleoperated  or  telepresence  systems. 


Fitts  (1954)  established  a  means  of  quantifying  human  performance  in 
terms  of  information  processing  capability  of  the  neororaclor  system.  He 
developed  a  relationship  between  the  speed  and  accuracy  of  human  motor 
performance  which  demonstrates  that  the  speed  of  a  task  requiring  a 
‘acy  is  bounded  by  the  capacity  of  the  neuromuscular  system 


control  movements.  Fitis  argued  that  if  manual  control  was  limited  by  the 
information  processing  rale  of  the  peripheral  and  ceniral  nervous  system, 
movement  times  would  be  limited  by  the  information  processing  demands  of 
the  task.  This  concept  is  apparent  if  we  think  of  a  task  requiring  great 
accuracy  such  as  threading  a  needle.  Why  do  we  perform  a  threading  task 
with  infuriating  sluggishness?  Such  a  task  is  limited  by  human  information 
processing  capacity  (i.e.,  how  fast  a  human  can  perceive  the  environment, 
actuate  limbs,  and  adjust  for  error).  A  task  such  as  threading  a  needle  requires 
many  fine  adjustments  and  is  thus  limited  by  how  many  cycles  of  perception 
and  adjustment  can  be  performed  per  unit  time  (i.e.,  limited  by  human 
bandwidth). 
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To  measure  human  information  processing  capacity  as  an  indication  of 
performance  within  the  telepresence  environment,  a  peg-in-hole  task  was 
used.  Like  threading  a  needle,  the  difficulty  of  peg  insertion  is  a  function  of 
the  difference  between  the  peg  diameter  and  the  hole  diameter.  The  tighter 
the  fit,  the  more  accuracy  required,  and  thus  the  slower  the  maximum 
performance  speed.  By  requiring  subjects  to  perform  a  standardized  peg 
insertion  task  as  fast  as  they  can,  completion  times  can  be  measured  and 
compared  to  task  difficulty  to  yield  information  processing  capacity  for  that 
task.  By  doing  such  an  analysis  upon  subjects  with  and  without  the  aid  of 
various  virtual  fixtures,  changes  in  task  completion  time  will  indicate 
performance  changes  resulting  from  fixture  use.  Thus,  by  using  a  peg-in- 
hole  Fitts'  Law  analysis,  the  effectiveness  of  various  virtual  fixtures  as 
perceptual  aids  can  be  quantified. 


Figure  1.  Measurements  Used  to  Define  Task  Difficulty 

Through  an  extensive  review  of  human  psychomotor,  perceptual,  and 
cognitive  test  batteries,  the  Naval  Oceans  Systems  Center  developed  a  peg-in- 
hole  performance  task  specifically  representative  of  teleoperator 
manipulative  activities  [22].  The  test  battery  requires  subjects  to  move  pegs 


of  various  diameter  between  holes  of  varied  spacing.  Movement  times  for  peg 
motions  are  recorded  and  correlated  with  task  difficulty.  As  defined  by  Fitts 
[8],  the  binary  Index  of  Difficulty,  ID,  for  the  one  dimensional  peg  transfer 
task  can  be  computed  as: 


2p^  (bits/response) 

where  A  is  the  amplitude  of  the  motion  and  W  is  the  peg  tolerance  defined  as 
the  difference  between  the  hole  diameter  and  the  peg  diameter.  These 
quantities  are  shown  in  Figure  1. 

Fitts’  Law  [8]  relates  task  completion  time  to  task  ID  by  defining  the 
movement  time,  mt,  as  follows: 


ID  =  -log2 


mt  =  k}  ID  +  k2  (sec) 

with  k  I  and  k2  being  characteristic  constants  of  the  individual  which 

represent  the  slope  and  intercept  of  the  Fitts'  Law  curve.  The  reciprocal  of 

the  slope  of  the  Fitts'  Law  curve  (1/kj)  has  units  (bils/sec)  which  are 

identical  to  units  of  capacity  for  an  information  channel  and  are  thus  an 
accepted  measure  of  human  information  processing  capacity  [25].  Because 
the  slope  is  susceptible  to  distortions  resulting  from  changes  in  strategy 
between  tasks  of  different  difficulty,  a  more  robust  measure  of  processing 
capacity  was  also  computed  called  the  binary  Index  of  Performance.  This 
value,  abbreviated  as  !p,  describes  the  information  processing  capacity 

required  of  the  operator  to  perform  a  task  of  a  particular  difficulty  [Fitts].  Ip 

has  units  (bits/sec)  and  is  defined  as  follows: 

.  1  ,  W 

log2  2P,  (bits/sec) 

ID 

Ip  =  ^^  (bits/sec) 

where  mt  is  the  movement  time  required  to  complete  the  task.  Index  of 
performance  Ip  is  an  accepted  measure  of  the  information  processing  capacity 
of  a  teleoperator  [25]. 


EXPERIMENTAL  HARDWARE 


Virtual  Fixture  Performance  Testbed 

To  implement  a  standardized  Fitts’  Law  performance  test  in  a 
telepresence  environment,  the  following  hardware  subsystems  were  used:  Peg 
Insertion  Task  Board,  MBA  Exoskeletal  Master  /  Merlin  Robot  Arm  Slave, 
Monocular  Vision  System,  and  Virtual  Fixture  Board. 


Figure  2.  Photograph  of  the  AF/Navy  Teleoperator  Performance 
Evaluation  Battery  Task  Board  Used  for  Implementing  Peg 
Insertion  Fitts’  Law  Task.  Also  shown  is  the  Merlin  Robot  Arm 
used  as  the  slave  in  the  leleoperaled  system. 

Task  Board:  AF/Navv  Teleoperator  Performance  Evaluation  Battery 

As  shown  in  Figure  2,  a  specialized  task  board  was  used  to  implement 
the  performance  task.  This  task  board  was  originally  developed  by  the  Naval 
Oceans  Systems  Center  to  implement  a  controlled  peg-in-hole  test  battery  for 
telemanipulation  systems  [22,  31].  Holes  on  the  board  are  arranged  to 

accommodate  a  wide  range  of  peg  sizes  and  movement  amplitudes.  This  study 
made  use  of  four  holes  (diameter  2.00  cm)  spaced  to  allow  horizontal  peg 
movements  of  4.00  cm  and  16.00  cm.  Three  pegs  (diameters  0.75  cm,  0.98  cm, 
1.50  cm)  were  used  in  the  task  to  vary  the  peg  insertion  tolerance.  The  bottom 


of  each  hole  contains  a  microswiich  connected  to  a  PC  via  a  Digital  I/O 
controller  card.  The  PC  controls  and  monitors  the  peg  insertion  portion  of  the 
experiment.  The  computer  is  equipped  with  a  real-time  clock  for  measuring 

peg  movement  times  to  the  nearest  millisecond.  The  timer  is  started  when  the 
peg  is  extracted  from  the  start  hole  and  stopped  when  inserted  into  the  target 
hole. 

MBA  Exoskeletal  Master  /  Merlin  Robot  Arm  Slave 

As  shown  in  Figure  3,  the  MBA  exoskeleton  master  is  a  dual-arm,  full 

body  exoskeleton  which  can  transduce  motion  in  seven  degrees  of 
freedom  for  each  arm  [16].  In  its  current  configuration,  the  device  is  used 

only  as  a  sensor;  no  force  information  is  reflected  back  to  the  user  through 
the  exoskeleton.  Thus,  peg  insertions  were  performed  based  only  on  visual 

and  auditory  feedback  from  the  workspace.  This  reduced  sensory  feedback 
environment  offered  a  good  testbed  for  the  prowess  of  virtual  fixtures. 


Figure  3.  Photograph  of  Subject  Wearing  MBA  Kinesthetic 
Exoskeleton  Used  as  the  Master  in  the  Teleoperator  System 


In  this  experiment,  the  position  of  the  subject’s  right  hand  was  tracked 
and  fed  to  the  slave  robot  arm.  An  end-effector  was  constructed  for  the  slave 
robot  which  housed  the  peg  and  allowed  for  easy  interchange  of  the  various 
diameter  pegs.  Because  of  the  difficulty  of  the  teleoperated  peg  insertion  task 
without  force  feedback,  the  slave  robot  was  programmed  to  maintain  its  end- 
effector  orientation  perpendicular  to  the  board  at  all  times.  This  arrangement 
simplified  the  task  and  eliminated  the  possibility  of  subjects  torquing  the  peg 
in  the  hole.  To  further  simplify  the  task,  the  robot  was  constrained  to  move 
only  in  a  horizontal  plane  level  with  the  target  holes.  Thus  the  task  only 
required  two-dimensional  positioning  of  the  peg  in  the  xy  plane  with  x 
defined  into  the  board,  and  y  defined  along  the  horizontal  row  of  target  holes 
as  shown  in  Figure  4.  It  should  be  noted  that  although  the  task  was 
constrained  to  planar  positioning,  it  was  sufficiently  difficult  to  require 
subjects  to  practice  for  two  45-minute  sessions  before  training  was  complete. 
The  task  was  designed  to  be  simple  in  concept  but  difficult  enough  that  the 
subjects  would  not  approach  the  upper  bound  of  the  slave  robot’s  ability  to 
perform  the  task. 


Figure  4.  Schematic  of  Task  Board  with  xy  Plane  of  Allowable  Peg  Motion 

To  protect  the  task  board  and  the  robot,  the  peg  was  rigidly  constrained 
in  all  degrees  of  freedom  except  along  the  x  axis  pointing  into  the  task  board. 
Along  that  axis,  the  peg  floated  on  a  stiff  spring  so  that  strong  forward 
impacts  of  the  peg  with  the  board  would  not  jar  the  board  or  damage  the  slave 
robot. 


Monocular  Vision  System 


A  previous  study  using  an  identical  task  board  performed  by  the  Naval 
Oceans  Systems  Center  has  shown  that  stereo  vision  feedback  offers  no 
performance  advantage  over  the  monocular  projections  used  in  this  peg 
insertion  task  [22,  31],  Thus,  a  simple  monocular  vision  system  was 

implemented  to  provide  visual  feedback  for  this  experiment.  The  system  was 
designed  as  an  inexpensive  means  of  creating  the  illusion  of  operator 
presence  while  providing  visual  feedback  of  the  workspace.  The  system  used 
7X  power  binocular  optics  focused  upon  a  distant  color  video  monitor  which 

displayed  the  output  of  a  single  camera  in  the  workspace  focused  on  the  task 

board.  The  monitor  was  placed  at  a  distance  from  the  optics  in  such  a  way  that 

the  magnification  of  the  video  image  created  the  illusion  that  the  task  board 
was  within  reach  of  the  operator’s  hands.  Robot  end-effector  motion  was 
scaled  to  match  operator  hand  motion  so  that  the  apparent  end-effector 

position  corresponded  to  the  user’s  kinesthetic  sense  of  arm  position.  Such  a 
correlation  between  kinesthetic  feedback  and  visual  feedback  of  robot  end- 

effector  position  greatly  enhanced  the  user’s  sense  of  presence  within  the 
workspace.  Testing  of  the  system  revealed  best  results  when  the  angle  of  the 

camera  incident  on  the  workspace  was  closely  matched  by  the  angle  that  the 

magnifying  optics  were  incident  upon  the  distant  video  monitor.  When  these 
angles  were  not  similar,  conflicting  perspective  cues  hindered  the  illusion  of 
presence.  Figure  5  includes  a  rough  schematic  of  the  vision  system  as  part  of 
the  overall  system  hardware. 

Virtual  Fixture  Board 


Rather  than  using  a  force  reflecting  exoskeleton  to  model  the  rigid 
impedance  surfaces  which  compose  the  virtual  fixtures,  it  was  thought  that 
the  preliminary  tests  of  virtual  fixtures  should  not  be  influenced  by 
hardware  limitations  of  force  reflecting  devices.  Thus  a  Fixture  Board  was 
designed  which  allowed  real  rigid  surfaces  to  be  developed  out  of  acrylic 
sheets.  The  MBA  exoskeleton  would  interact  with  the  real  acrylic  surfaces  and 
reflect  that  information  to  the  user.  As  far  as  the  user  was  concerned,  the 
haptic  perceptions  of  these  surfaces  were  coming  from  the  exoskeleton  and 
were  just  as  much  ’’virtual”  information  as  if  they  were  truly  computer 


generated.  The  benefit  of  this  approach  was  that  the  reflected  perceptions  of 
these  surfaces  were  perfectly  modeled.  The  surfaces  felt  crisp  and  real,  free 
from  the  bandwidth  limitations  of  most  force  reflecting  devices.  The  drawback 
of  this  approach  was  that  more  abstract  fixtures  were  impossible  to  generate 
using  this  method. 


Figure  5.  Complete  Telepresence  System  Developed  to  Implement 
the  Testing  of  Teleoperator  Performance  in  a  Standardized  Peg 
Insertion  Task  with  and  without  the  Aid  of  Virtual  Fixtures 

The  virtual  fixtures  were  constructed  out  of  acrylic  sheets  and 

positioned  upon  the  fixture  board,  a  wooden  platform  positioned  in  front  of 

the  user  as  shown  in  Figure  5.  To  make  the  fixtures  quickly  interchangeable, 

they  were  constrained  and  positioned  on  the  wooden  platform  by  locating 

pins.  The  exoskeleton  was  fitted  with  a  Teflon  cap  at  the  end  of  the  hand  grip 
which  was  used  as  the  contact  surface  between  the  exoskeleton  and  the  fixture. 
The  fixtures  were  treated  regularly  with  an  oil-based  lubricant  so  that  little 
friction  was  noticeable  between  the  Teflon  and  the  acrylic  surfaces. 
Although  frictional  surfaces  could  make  for  effective  virtual  fixtures,  the 


intent  of  this  study  was  to  look  at  very  simple  surfaces  modeled  only  as  rigid 
impedances  free  from  any  viscous  or  coulombic  damping. 

Although  the  haptic  fixtures  were  modeled  physically  rather  than 
computationally  5  auditory  fixtures  were  tested  that  were  pure  computer 
simulations.  Simple  compliant  surfaces  were  modeled  in  which  the 
compression  of  the  surface  was  proportional  to  a  linear  change  in  pitch. 
Tones  were  generated  on  a  PC  and  fed  to  the  user  via  stereo  headphones.  These 
trials  served  as  a  testoed  for  alternate  modality  fixtures  as  well  as  a  proof  of 
concept  of  purely  computer-generated  fixtures. 

Overall  Test  Setup 


Having  described  each  of  the  major  components  of  this  test  setup,  it  is 
important  to  clearly  describe  the  system  as  a  whole  before  getting  into  the 

details  of  the  subject  testing.  As  shown  in  Figure  5,  the  system  is  divided  into 
two  physically  separate  parts:  the  remote  environment  and  the  operator 

space.  The  remote  environment  contains  the  task  board,  the  Merlin  robot  arm, 

and  a  single  video  camera  pointed  at  the  task  board.  The  camera  is  positioned 
so  that  the  incident  perspective  is  similar  to  what  a  human  operator  would  see 
if  standing  directly  in  front  of  the  board  and  performing  the  peg  insertions  in 
person.  The  operator  space  contains  the  exoskeleton,  the  vision  system,  and 
the  virtual  fixture  board.  Once  inside  the  exoskeleton  and  vision  system,  the 
subject  is  presented  with  a  projection  of  the  image  from  the  camera  in  the 
remote  environment.  The  subject  is  given  the  illusion  that  the  task  board  is 

situated  directly  before  him,  within  reaching  distance  of  the  exoskeleton.  In 

reality,  the  task  board  is  on  the  opposite  side  of  the  laboratory,  behind  the 

subject  and  completely  out  of  view.  The  fixture  table  is  placed  directly  in  front 
of  the  subject  in  such  a  way  that  it  cannot  be  seen  when  looking  through  the 

vision  system,  but  feels  as  though  it  occupies  the  same  space  as  the  apparent 
image  of  the  task  board.  Thus,  virtual  fixtures  implemented  on  the  fixture 

board  feel  as  though  they  are  overlaid  on  top  of  their  perception  of  the  remote 


environment. 


The  remote  robot  arm  is  slaved  to  the  right  hand  position  of  the 

exoskeleton.  Thus,  when  the  subject  moves  his  right  hand  to  interact  with  the 
image  of  the  task  board,  the  end-effector  on  the  slave  arm  follows.  Because  the 
subject  cannot  see  his  own  hand  when  looking  into  the  vision  system  but  does 
see  the  remote  robot  end-effector  in  the  position  where  he  feels  his  hand  to 
be,  a  sense  of  presence  within  the  remote  environment  is  created.  The 

subjects  also  wear  a  set  of  stereo  headphones  for  use  in  implementing  auditory 
perceptual  overlays.  Auditory  surfaces  and  fields  can  be  modeled  on  the 

control  computer  and  interacted  with  by  the  user.  The  complete  system 
provides  a  powerful  testbed  for  projecting  subjects  into  a  remote  environment 
and  overlaying  haptic  and  auditory  information  on  top  of  the  reflected 
percepts.  The  set  of  tests  described  in  this  paper  implements  only  haptic 
sensory  overlays,  although  a  previous  study  implemented  both  haptic  and 

auditory  virtual  fixtures  [26]. 


EXPERIMENTAL  DESIGN  &  PROCEDURE 


Experimental  Design 

The  goal  of  this  study  was  to  investigate  the  effect  that  the  presentation 
of  overlaid  virtual  fixtures  has  upon  performance  in  a  standard  teleoperated 
task  with  time  delay.  A  previous  study  compared  eight  virtual  fixture  designs 
in  a  telemanipulation  task  without  time  delay  [Rosenberg].  For  this 

experiment,  three  of  the  eight  previously  tested  fixtures  were  used  for 
comparison  to  a  no-fixture  control  case.  Each  of  the  three  fixture 

configurations  was  tested  with  no  delay,  250  ms  delay,  and  450  ms  latency 
between  master  and  slave.  Figure  6  shows  a  schematic  representation  of  each 
of  the  fixtures  tested.  (NOTE:  The  fixtures  are  labeled  with  the  same  numbers 

given  in  the  previous  study  for  reference  purposes.)  These  fixtures  are  shown 
overlaid  on  top  of  the  fixture  board  as  they  are  perceived  by  the  subjects. 
Although  these  fixtures  are  represented  graphically  in  this  figure,  they  are 

perceived  only  as  forces  by  the  subjects.  Although  instilling  visual  qualities 
to  fixtures  is  a  viable  application  of  perceptual  overlays  and  an  important 
topic  for  future  investigation  into  virtual  fixtures,  this  study  was  restricted 
only  to  haptic  modalities. 

The  three  virtual  fixtures  tested  are  composed  of  simple  combinations  of 
planar  surfaces  which  are  perceived  haptically  as  rigid  wall  sensations  by 

the  users.  As  shown  in  Figure  6,  virtual  Fixture  4  is  composed  of  three 
intersecting  planar  surfaces.  One  surface  is  simply  a  horizontal  plane 
oriented  like  a  table  top  in  the  workspace  and  aligned  so  that  contact  with  the 
rigid  surface  will  result  in  vertical  alignment  of  the  operator’s  hand  with  the 
holes  in  the  peg  board.  The  second  plane  runs  diagonally  across  the  first 
plane  in  such  a  way  that  interaction  with  this  rigid  surface  will  guide  the 
subject’s  hand  motion  across  the  face  of  the  board  and  inward  towards  the 
target  hole.  The  third  plane  runs  parallel  with  the  target  hole  such  that 
contact  with  the  plane  will  halt  hand  motion  when  the  peg  passes  in  front  of 
the  target  hole. 
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TOP  VIEW 


NO  FIXTURE 


FIXTURE  4 


TOP  VIEW 


FIXTURE  5 


TOP  VIEW 


FIXTURE  6 


Figure  6.  Virtual  Fixtures  Shown  Overlaid  onto  Workspace  as 
They  are  Perceived  by  Subjects.  Also  shown  are  top  views, 
looking  down  from  above  the  task  board.  These  fixtures  are 
shown  graphically  here,  but  they  are  perceived  haptically  by 
subjects. 


To  clarify  the  implementation  of  Fixture  4,  Figure  7  shows  a  sample 
trajectory  superimposed  upon  the  schematic  diagram  of  the  fixture.  This 
drawing  demonstrates  how  the  use  of  a  fixture  might  influence  the  trajectory 
of  the  operator’s  hand.  As  shown  in  the  figure,  the  rigid  surfaces  do  not 
define  the  exact  motion  of  the  peg,  but  rather  influence  the  trajectory  by 
imposing  rough  kinematic  boundaries  on  operator  hand  motion. 


Figure  7.  Sample  Trajectory  of  Peg  from  Start  Hole  to  Target  Hole 
as  Guided  by  a  Virtual  Fixture.  As  shown,  the  fixture  does  not 
define  the  motion  of  the  operator’s  hand,  but  rather  defines  the 
boundaries  of  the  motion. 

Virtual  Fixture  6  is  similar  to  virtual  Fixture  4  except  that  the  surface 

which  stops  hand  motion  in  front  of  the  target  hole  is  removed  and  two 
diagonal  surfaces  are  added  in  its  place.  These  two  diagonal  surfaces  are 

positioned  and  oriented  to  assist  in  the  fine  positioning  of  the  peg  into  the 
target  hole. 

Virtual  Fixture  5  is  very  different  from  the  two  fixtures  presented  thus 

far  in  both  its  geometry  and  implementation.  Fixture  5  is  a  rigid  surface 
oriented  parallel  to  the  task  board  and  located  approximately  0.5"  in  front  of 
the  board.  This  fixture  was  not  designed  for  interaction  with  the  subject's 
right  hand  (the  hand  that  performed  the  peg  insertion  task),  but  rather  was 
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designed  for  interaction  with  the  subject’s  unused  left  hand.  Subjects  were 

told  to  place  their  left  hand  upon  the  surface  while  performing  the  task  with 
the  right  hand.  The  intent  of  this  fixture  is  to  isolate  the  effect  of  localization 
on  performance  in  lime  delayed  teleoperation  tasks.  A  previous  study  of 
virtual  fixtures  in  teleoperation  without  time  delay  suggested  that  interaction 
with  virtual  fixtures  provides  localizing  information  to  the  user  which  can 
enhance  understanding  of  the  remote  workspace.  It  was  hypothesized  that  the 
localization  provided  by  virtual  fixtures  allows  for  better  correlation  of  the 
operator's  kinesthetic  sense  of  hand  position  to  the  geometry  of  the  remote  site 
[26].  It  is  desired  now  to  investigate  if  enhanced  localization  alone  can  reduce 
the  detrimental  effects  that  time  delay  has  upon  telemanipulation.  Because  the 
other  fixtures  to  be  tested  in  this  study  act  to  guide  the  operator's  hand  motion 

as  well  as  to  provide  localizing  references  to  the  remote  workspace,  it  would 
be  impossible  to  isolate  the  effects  of  the  localizing  information  alone.  Fixture 
5  was  therefore  designed;  it  only  interacts  with  the  unused  hand  and  thus  can 

only  have  a  localizing  effect  upon  operator  performance. 

Experimental  Protocol 

A  series  of  tests  was  run  to  evaluate  subject  performance  using  each 

fixture  configuration  for  each  of  three  time  delay  periods:  no  delay,  250  ms 
delay,  and  450  ms  delay.  Operator  performance  was  recorded  during  test 

periods  which  included  12  practice  and  36  timed  peg  insertion  trials.  A 

single  trial  consisted  of  moving  a  peg  from  a  designated  start  hole  to  a 
designated  target  hole.  The  holes  were  referenced  by  numerals  (3,  4,  5  or  6) 
located  above  each  hole,  as  shown  in  Figure  4.  Two  start  hole/target  hole  pairs 
were  studied  in  these  tests:  a  16  cm  motion  from  hole  3  to  hole  6  and  a  4  cm 
motion  from  hole  5  to  hole  6.  The  36  trial  period  was  divided  into  three  groups 
of  12  trials.  Each  of  these  groups  required  the  subject  to  perform  the  insertion 
task  using  a  different  peg  size.  After  the  completion  of  each  group  of  12  trials, 
subjects  were  instructed  to  rest  for  approximately  two  minutes  while  one  of 
three  possible  peg  sizes  was  installed  in  the  robot  end-effector. 

The  use  of  two  motion  amplitudes  (4  cm  and  16  cm)  and  three  peg  sizes 

(0.75  cm,  0.98  cm,  1.50  cm  diameter  pegs)  allowed  for  the  testing  of  insertion 

trials  with  six  different  task  difficulties.  Table  1  shows  all  combinations  of 
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peg  size  and  motion  amplitude  and  lists  the  Index  of  Difficulty  for  each  task  as 
dictated  by  Fitts’  Law  [8],  The  order  in  which  the  three  peg  sizes  were 
presented  to  each  subject  was  randomized  to  ensure  that  mental  and  physical 
fatigue  had  similar  effects  on  all  task  difficulties. 


Table  1.  Index  of  Difficulties  (ID)  Shown  for  All  Combinations  of 
Peg  Size  and  Peg  Motion  Amplitudes  as  Predicted  by  Fitts’  Law 


Motion  Amplitude 


Diameter 

5  to  6  (4  cm) 

3  to  6  (16cmj  ! 

0.75  cm 

2.68  bits/response 

4.68  bits/response 

0.98  cm 

2 ‘98  bits/response 

4,98  bits/response 

1.50  cm 

4.00  bits/response 

6.00  bits/response 

Subjects  were  instructed  to  begin  each  test  period  with  a  3  to  6  peg 
insertion  trial.  Subjects  would  then  perform  a  5  to  6  peg  insertion  trial,  then 

a  3  to  6  peg  insertion  trial,  and  cycle  in  that  manner  throughout  the  36  trials. 
Subjects  were  allowed  to  proceed  through  the  test  period  at  their  own  pace.  To 

automate  the  testing  procedure  and  allow  subjects  to  proceed  through  the  test 

with  little  operator  intervention,  a  number  of  simple  beeps  was  used  to  guide 

subject  activity.  When  a  subject  inserted  a  peg  into  the  correct  starting  hole, 
the  task  board  control  computer  would  emit  an  audible  tone  to  signal  that  the 

peg  was  properly  positioned.  The  subject  would  then  keep  the  peg  in  the  start 
hole  for  two  seconds  until  the  control  computer  produced  a  second  tone  which 

signaled  that  the  task  could  now  be  performed  at  will.  The  reason  for  this  two 

second  waiting  period  was  to  ensure  that  the  subject  maintained  a  steady 

contact  with  the  microswitch  at  the  base  of  the  start  hole.  The  subjects  were 

not  required  to  perform  the  insertion  task  as  soon  as  the  second  lone  was 

heard;  this  second  tone  simply  meant  "you  are  free  to  go  whenever  ready." 
Thus,  the  subjects  could  mentally  prepare  themselves  for  the  particular 
insertion  task  and  begin  at  will.  When  ready,  the  subject  would  remove  the  peg 
from  the  start  hole  and  insert  it  into  the  target  hole  as  fast  as  possible.  Upon 

insertion  into  the  target  hole,  the  control  computer  would  emit  a  tone  to  signal 
that  the  trial  had  been  successfully  completed.  The  subject  was  then  free  to 


proceed  to  the  next  insertion  trial  at  will.  At  the  end  of  each  trial  the  control 
computer  recorded  the  movement  time  along  with  the  peg  size  and  motion 

amplitude.  After  the  completion  of  each  group  of  12  trials,  the  control 
computer  would  emit  a  long  duration  tone  which  signaled  the  subject  to  rest 
for  two  minutes  while  the  operator  exchanged  peg  sizes.  Post  testing 

interviews  revealed  that  all  subjects  were  comfortable  with  the  use  of  audible 
tones  to  automate  the  testing  procedure. 

Subjects  were  tested  over  6  experimental  sessions,  each  lasting  45  to  60 
minutes.  To  minimize  the  effects  of  fatigue  and  boredom,  no  subject  completed 
more  than  one  experimental  session  during  a  single  day  of  testing.  The  first 
two  experimental  sessions  included  three  test  periods  of  36  trials.  The  first 

period  of  the  session  was  performed  with  no  fixture  and  no  delay,  the  second 

was  performed  with  no  fixture  and  250  ms  delay,  and  the  third  was  performed 
with  no  fixture  and  450  ms  delay.  These  initial  216  trials  were  treated  only  as 
practice  during  which  the  subjects  familiarized  themselves  with  the  use  of  the 

exoskeleton,  Merlin  robot  arm,  vision  system,  and  fixture  table.  It  was  also 
intended  that  subjects  get  experience  with  the  effect  that  time  delay  has  upon 

telemanipulation. 

It  was  found  that  by  the  end  of  the  second  practice  session,  all  subjects 
had  sufficiently  learned  the  task  that  variability  in  movement  times  for  trials 
of  the  same  difficulty  had  fallen  below  20%  for  every  subject,  with  a  mean 
variability  of  15%  for  all  subjects.  Once  learning  had  stabilized,  subjects  were 
sequentially  tested  using  each  of  the  test  fixtures  for  each  of  three  time  delay 
scenarios.  A  new  fixture  configuration  was  tested  during  each  of  the  next 
four  daily  sessions.  Table  2  lists  all  of  the  daily  sessions  along  with  the  fixtures 
tested  during  that  session.  The  order  of  the  last  four  daily  sessions  was 
randomized  for  each  subject  to  reduce  the  effect  that  learning  might  have 
upon  performance  results. 
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Table  2.  Testing  Schedule:  All  Four  Tests  Shown  along  with 
Corresponding  Daily  Sessions  and  Fixtures  Tested  on  that  Day 


TEST  I  SESSION 


FIXTURES  STUDIED 


Practice 

day  1 

No  Fixture: 

no  delay  250ms  delay  450ms  delay 

Practice 

day  2 

No  Fixture: 

no  delay  250ms  delay  450ms  delay 

Test  I 

day  3 

No  Fixture: 

no  delay  250ms  delay  450ms  delay 

Test  II 

day  4 

Fixture  5: 

no  delay  250ms  delay  450ms  delay 

Test  III 

day  5 

Fixture  4: 

no  delay  250ms  delay  450ms  delay 

Test  IV 

day  6 

Fixture  6: 

no  delay  250ms  delay  450ms  delay 

£ 


RESULTS 


The  following  section  presents  the  results  of  peg  insertion  trials  for 
each  time  delay  period  for  comparison  among  each  of  the  virtual  fixture 

configurations  tested.  The  basic  quantities  presented  are  the  recorded 
movement  times  for  peg  insertion  trials  and  the  coefficient  of  variation  (COV) 
across  movement  times.  The  COV  has  been  computed  across  the  trials  of  the 

same  index  of  difficulty  performed  with  the  same  time  delay  and  same  virtual 

fixture  configuration.  The  COV  is  therefore  a  measure  of  subject  consistency 
in  performing  a  particular  task  with  a  particular  latency  and  using  a 

particular  virtual  fixture.  Tables  3  through  6  list  the  mean  movement  time 
results  across  all  six  subjects  for  each  Index  of  Difficulty  and  each  time  delay 
configuration.  Table  7  lists  the  coefficients  of  variation  in  movement  times 
for  each  time  delay  and  fixture  configuration  tested. 

Table  3.  Mean  Movement  Times  for  Each  Index  of  Difficulty  are  Shown  for  Six 
Subjects  using  NO  FIXTURE  to  Perform  Peg  Insertion  Tasks  with  No  Delay,  250  ms 
Delay,  and  450  ms  Delay.  Also  computed  is  the  mean  movement  time  across  all 
trials  (across  task  difficulty). 

TEST  1:  no  fixture  trials _ 

INDEX  OF  DIFFICULTY 

DELAY  2.68  |2.98  |4.00  |4.68  |4.98  |6.00  mea^ 

no  delay  833ms  927ms  991ms  1617ms  1600ms  1666ms  1272ms 

250ms  1144ms  1246ms  1374ms  2123ms  2142ms  2362ms  1731ms 

450ms  1165ms  1284ms  1500ms  2140ms  2329ms  2636ms  1842ms 
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Table  4.  Mean  Movement  Times  for  Each  Index  of  Difficulty  arc  Shown  for  Six 
Subjects  using  FIXTURE  5  to  Perform  Peg  Insertion  Tasks  with  No  Delay,  250  ms 
Delay,  and  450  ms  Delay.  Also  computed  is  the  mean  movement  time  across  all 
trials  (across  task  difficulty). 


TEST  II:  fixture  5  trials 


DELAY 

2.68 

IND 

2.98 

EX  OF  I 

4.00 

3IFFICU1 

4.68 

OO 

6.00 

mean 

no  delay 

656ms 

662ms 

807ms 

14  4  8ms 

1396ms 

1497ms 

1077ms 

250ms 

819ms 

882ms 

97  5ms 

1683ms 

1814ms 

1892ms 

1344ms 

450ms 

84  6ms 

1003ms 

1181ms 

1941ms 

2015ms 

2273ms 

1543ms 

Table  5.  Mean  Movement  Times  for  Each  Index  of  Difficulty  are  Shown  for  Six 
Subjects  using  FIXTURE  4  to  Perform  Peg  Insertion  Tasks  with  No  Delay,  250  ms 
Delay,  and  450  ms  Delay.  Also  computed  is  the  mean  movement  time  across  all 
trials  (across  task  difficulty). 


TEST  III:  fixture  4  trials 


DELAY 

2.68 

IND 

2.98 

EX  OF  1 

4.00 

31FF1CU1 

4.68 

LTY 

4.98 

6.00 

mean 

no  delay 

679ms 

682ms 

7  91ms 

1375ms 

1467ms 

150  6ms 

1083ms 

250ms 

7  62ms 

757ms 

95  6ms 

1473ms 

1547ms 

1625ms 

1187ms 

450ms 

696ms 

596ms 

7  4  1ms 

1518ms 

1513ms 

1540ms 

1100ms 

Table  6.  Mean  Movement  Times  for  Each  Index  of  Difficulty  are  Shown  for  Six 
Subjects  using  FIXTURE  6  to  Perform  Peg  Insertion  Tasks  with  No  Delay,  250  ms 
Delay,  and  450  ms  Delay.  Also  computed  is  the  mean  movement  time  across  all 
trials  (across  task  difficulty). 


TEST  IV:  fixture  6  trials 


DELAY 

2.68 

IND 

2.98 

EX  OF  1 

4.00 

3IFFICU1 

4.68 

LTY 

4.98 

6.00 

mean 

no  delay 

641ms 

691ms 

811ms 

12  66ms 

1361ms 

1461ms 

1039ms 

250ms 

618ms 

669ms 

717ms 

1 4  89ms 

1417ms 

1579ms 

1082ms 

450ms 

635ms 

591ms 

721ms 

1365ms 

14  99ms 

1539ms 

1058ms 
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Table  7.  Mean  Coefficient  of  Variation  for  All  Subjects  Performing  Tasks  of  the 
Same  Difficulty  with  Each  Fixture  as  Shown.  Also  shown  is  the  overall  COV 
which  indicates  the  overall  consistency  of  the  results. 


NO  RXTURE 

FIXTURE  5 

FIXTURE  4 

FIXTURE  6 


DELAY 

COV 

No  delay 

14%  ±3.3 

250  ms 

14%  ±3.2 

450  ms 

18%  ±3.9 

No  delay 

13%  ±1.3 

250  ms 

13%  ±1.7 

450  ms 

17%  ±2.7 

No  delay 

8%  ±1.5 

250  ms 

9%  ±1.0 

450  ms 

14%  ±1.8 

No  delay 

10%  ±2.8 

250  ms 

14%  ±2.0 

450  ms 

12%  ±3.0 

All  Trials 


In  order  to  quantify  the  drop  in  operator  performance  resulting  from 
the  introduction  of  time  delays  into  the  telcmanipulation  task  in  a  way  that 

could  be  easily  compared  among  the  various  virtual  fixture  configurations 
tested,  percentage  increase  in  movement  time  was  computed  as  a  unitiess 
measure  of  the  effect  that  time  delay  had  upon  operator  performance. 
Performance  degradation  was  thus  quantified  as  the  percentage  increase  in 
movement  time  for  peg  insertion  trials  with  either  250  ms  and  450  ms  delays  as 

compared  to  the  trials  with  no  time  delay.  Tables  8  through  11  list  the 

percentage  increase  in  movement  times  corresponding  to  the  use  of  each 

virtual  fixture  tested.  Figure  8  shows  the  mean  percentage  increase  in 

movement  time  associated  with  each  time  delay  configuration  for  all  of  the 
virtual  fixtures  tested  as  well  as  for  the  no-fixture  case. 
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Table  8.  Performance  Degradation  Due  to  Time  Delays  for  All  Six  Subjects 
Performing  Task  using  NO  FIXTURE.  Performance  degradation  computed  as 
percentage  increase  in  movement  times  for  peg  insertion  trials  v/ith  250  ms 
and  450  ms  delays  as  compared  to  the  trials  with  no  time  delay. 


TEST  I:  no  fixture  trials 


DELAY 

2.68 

INDE 

2.98 

X  OF  Dll 

4.00 

'FICULT' 

4.68 

{ 

4.98 

6.00 

mean 

stdev 

250ms 

37.3  % 

1  34 .4  % 

38.6  % 

31.3  % 

33.9  % 

41.8  % 

36  % 

±10.8 

450ms 

39.9  % 

[38.5  % 

51.4  % 

32~1  % 

45.6  % 

58.2  % 

IT . 

TT  ?  .  4  I 

Table  9.  Performance  Degradation  Due  to  Time  Delays  for  All  Six  Subjects 
Performing  Task  using  FIXTURE  5.  Performance  degradation  computed  as 
percentage  increase  in  movement  times  for  peg  insertion  trials  with  250  ms 
and  450  ms  delays  as  compared  to  the  trials  with  no  time  delay. 


TEST  II:  fixture  5  trials 


INDEX  OF  DIFFICULTY 

DELAY 

2.68 

2.98 

4.00 

4.68 

4.98 

6.00 

mean 

Stdev 

250ms 

24.9  % 

33.3  % 

20.8  % 

16.2  % 

30.0  % 

26.4  % 

25  % 

±  9.5 

450ms 

28.9  % 

51.5  % 

46.3  % 

34.0  % 

51.8  % 

43  % 

±17.8 

Table  10.  Performance  Degradation  Due  to  Time  Delays  for  All  Six  Subjects 
Performing  Task  using  FIXTURE  4.  Performance  degradation  computed  as 
percentage  increase  in  movement  times  for  peg  insertion  trials  with  250  ms 
and  450  ms  delays  as  compared  to  the  trials  with  no  time  delay. 


TEST  III:  fixture  4  trials 


INDEX  OF  DIFFICULTY 

DELAY 

2.68 

2.98 

4.00 

4.68 

4.98 

6.00 

mean 

Stdev 

250ms 

12.2  % 

11.1  % 

20.8  % 

7.1  % 

5.5  % 

7.9  % 

10  % 

±  2.9 

450ms 

2.5  % 

-12.6  % 

-6.3  % 

10.4  % 

3.2  % 

2.3  % 

2  % 

±  6.8 
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Table  11.  Performance  Degradation  Due  to  Time  Delays  for  All  Six  Subjects 
Performing  Task  using  FIXTURE  6.  Performance  degradation  computed  as 
percentage  increase  in  movement  times  for  peg  insertion  trials  with  250  ms 
and  450  ms  delays  as  compared  to  the  trials  with  no  time  delay. 


TEST  IV:  fixture  6  trials 


INDEX  OF  DIFFICULTY 

2.68 

2.98 

4.00 

4.68 

4.98 

6.00 

mean 

stdev  j 

250  ms 

-3.7  % 

-3.3  % 

-11.6  % 

17.6  % 

4.13  % 

8,1  % 

±  10.1 

450  ms 

-1.0  % 

-14.6  % 

-11.1  % 

7.9  % 

10,1  % 

5.4  % 

mm 

±  8.9 

Psrforoianee  Degraciation  Dye  to  Time  Delay 


12  3  4 

NO  FIXTURE  FIXTURE  5  FIXTURE  4  FIXTURE  6 


Figure  8.  Performance  Degradation  Due  to  Time  Delay  for  Each  of  the 
Four  Virtual  Fixture  Configurations  Tested.  Performance  degradation 
computed  as  mean  percentage  increase  in  movement  time  for  peg 
insertion  trials. 


DISCUSSION 


Overview  of  Results 

The  following  section  addresses  the  implications  of  performance 
degradation  due  to  time  delay  and  investigates  the  effect  that  virtual  fixturing 
has  upon  operator  performance  in  time  delayed  teleopcration.  The  objective 
of  this  analysis  is,  first,  to  quantify  the  effect  that  time  delay  has  upon 

operator  performance  in  a  controlled  telemanipulation  task  and,  second,  to 
determine  if  virtual  fixtures  can  be  used  to  effectively  reduce  the  resulting 
drops  in  operator  performance.  Performance  degradation  has  been  computed 
as  the  percentage  increase  in  movement  time  for  a  task  of  given  difficulty 
when  a  particular  time  delay  is  introduced  between  the  operator  and  the 

remote  environment  as  compared  to  the  no-dclay  case. 

Before  comparing  the  drops  in  operator  performance  associated  with 
the  introduction  of  time  delay  to  the  interface,  the  issue  of  repeatability  and 

reliability  of  the  results  is  addressed.  As  shown  in  Table  7,  the  coefficient  of 
variation  in  movement  times  for  trials  with  a  given  time  delay  and  fixture 
configuration  is  under  18%  for  all  combinations  of  test  parameters.  The  mean 

coefficient  of  variation  is  13%  across  all  test  parameters.  This  suggests  that  for 
a  given  task  of  a  particular  difficulty,  performed  by  a  given  subject,  with  a 

particular  time  delay  and  fixture  configuration,  the  variation  in  performance 

among  all  such  trials  was,  on  average,  13%.  Such  consistency  in  human 
performance  testing  is  surprisingly  good  and  suggests  that  the  results  of  the 
experimental  sessions  are  highly  reliable  and  repeatable. 

Peg  Insertion  Trials  Without  Virtual  Fixture 

As  listed  in  Table  8  and  shown  in  Figure  8,  the  time  required  to 

successfully  complete  a  peg  insertion  trial  was  found  to  increase  dramatically 
when  time  delays  were  introduced  into  the  telemanipulation  interface.  For 
trials  with  small  time  delays  of  250  ms  and  450  ms,  movement  times  increased, 
on  average,  by  36%  and  44%  respectively.  This  dramatic  increase  in  movement 

times  resulting  from  the  introduction  of  small  time  delays  suggests  that 
operator  performance  in  even  the  most  simple  telemanipulation  tasks  is 
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highly  susceptible  to  degradation  due  to  time  delays.  Assuming  an  ideal 
lelemanipulation  system  with  no  time  lag  due  to  information  processing  and 
computation,  time  delays  of  250  ms  and  450  ms  would  correspond  to 
communication  transmission  lags  that  would  occur  if  the  operator  and  the 
remote  site  were  separated  by  distances  as  small  as  37000  km  and  67500  km 
respectively.  Such  distances  would  be  typical  when  controlling  a  remote 
manipulator  in  geosynchronous  earth  orbit  from  a  command  station  on  the 
ground  [13],  The  results  of  this  study  suggest  that  to  achieve  efficient 
telemanipulation  in  earth  orbit,  some  means  of  reducing  performance 

degradation  due  to  communication  transmission  lags  is  required.  In  a  non¬ 
ideal  telemanipulation  system,  latencies  due  to  information  processing  and 
computation  would  compound  the  total  time  delay  and  reduce  performance 
even  more. 


Peg  Insertion  Trials  With  Virtual  Fixtures 

As  listed  in  Table  9  through  Table  11  and  shown  in  Figure  8,  the 

increases  in  the  movement  time  to  successfully  complete  a  peg  insertion  trial 
when  virtual  fixtures  were  used  was  significantly  less  than  the  increases  in 
movement  times  for  unaided  trials.  Turning  first  to  peg  insertion  trials  using 

virtual  Fixture  5,  Figure  6  shows  that  Fixture  5  is  composed  of  a  single  rigid 
impedance  plane  which  is  parallel  to  the  task  board  surface  and  located 

approximately  0.5"  in  front  of  the  board.  Whereas  the  other  two  fixtures  tested 
were  designed  for  interaction  with  the  operator’s  right  hand  (the  hand  that 
manipulates  the  peg),  Fixture  5  is  designed  for  interaction  only  with  the 
operator's  unused  left  hand.  Subjects  place  the  palm  of  their  left  hand  upon 

the  surface  while  they  perform  the  task  with  their  right  hand.  This  fixture 
was  tested  to  isolate  the  effect  of  localization  upon  performance.  As  shown  in 
Figure  8,  the  increases  in  movement  times  associated  with  time  delays  of  250 
ms  and  450  ms  were  25%  and  43%  respectively.  Comparing  this  to  the  36%  and 
44%  increases  in  unaided  trials,  we  see  that  the  use  of  Fixture  5  resulted  in  a 
small  drop  in  performance  degradation  for  the  250  ms  delay  trials  but  no  drop 
in  performance  degradation  for  the  450  ms  delay  trials.  This  result  suggests 
that  a  single  localizing  reference  percept  presented  in  isolation  does  play  a 
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time  delay,  but  that 


smaii  part  in  reducing  the  performance  degradation  due  to 
more  elaborate  fixtures  are  likely  required  to  significantly 
detrimental  effects  of  Umc  delay  upon  performance. 


combat  the 


Turning  next  to  peg  insertion  trials  performed  with  the  aid  of  virtual 

Fixture  4,  we  recall  that  Fixture  4  is  composed  of  three  intersecting  rigid 
surfaces  as  shown  in  Figure  6.  One  is  a  horizontal  surface  oriented  like  a 
table  top  in  the  workspace  and  aligned  so  that  contact  v/ith  the  surface  will 
result  in  vertical  alignment  of  the  operator’s  hand  with  the  holes  in  the  peg 
board.  The  second  plane  runs  diagonally  across  the  first  such  that  interaction 
v/ith  this  surface  will  guide  a  subject’s  hand  across  the  face  of  the  board  and 
inv/ard  towards  the  target  hole.  The  third  plane  runs  parallel  with  the  target 

hole  such  that  contact  with  the  plane  will  halt  hand  motion  when  the  peg  is 
in  front  of  the  target  hole.  Remember  that  the  fixture  surfaces  are  perceived 
as  overlaid  haptic  sensations  only;  subjects  cannot  see  the  fixtures  when 

performing  the  task.  It  should  also  be  noted  that  interaction  with  the  overlaid 
haptic  sensations  provides  real-time  sensory  feedback  to  the  operator  even 

though  all  interaction  with  the  remote  site  is  time  distorted. 


When  provided  with  the  real-time  haptic  information  from  virtual 
Fixture  4,  performance  degradation  in  the  time  delayed  telcmanipulation  task 
was  significantly  less  than  in  the  no-fixture  trials.  As  listed  in  Table  10  and 
shown  in  Figure  8,  peg  insertion  trials  using  virtual  Fixture  4  with  lime 
delays  of  250  ms  and  450  ms  resulted  in  movement  lime  increases  of  10%  and 
2%  respectively.  Comparing  this  result  to  the  36%  and  44%  movement  time 
increases  for  the  no-fixturc  trials,  it  is  clear  that  the  use  of  virtual  Fixture  4 
has  greatly  reduced  the  drop  in  performance  resulting  from  the  introduction 
of  time  delays  into  the  telcmanipulation  interface.  In  fact,  for  the  longer 
delay  of  450  ms,  the  use  of  virtual  Fixture  4  results  in  movement  times  that 
reflect  no  statistically  significant  drop  in  operator  performance  as  compared 
to  the  no-delay  trials.  The  fact  that  less  performance  degradation  was  recorded 
for  the  450  ms  trials  than  was  recorded  for  the  250  ms  trials  implies  that 
subjects  had  an  easier  time  taking  advantage  of  the  real-time  sensory 
information  provided  by  the  virtual  fixture  in  the  trials  with  the  longer 
delay.  In  post  testing  interviews,  subjects  reported  finding  it  easier  to  focus 
their  concentration  on  the  real-time  haptic  sensations  provided  by  the  virtual 
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fixtures  and  focus  less  on  the  lime  delayed  feedback  from  the  remote  site 
when  the  delay  period  was  longer.  In  psychophysical  terms,  when  the  latency 
is  short,  the  time  delayed  sensory  information  competes  with  real-time 
sensory  information  provided  by  the  fixtures  and  masks  their  effects.  The 
shorter  the  delay,  the  more  the  real-time  and  delayed  sensations  compete,  and 
the  greater  the  masking.  This  result  is  consistent  with  many  psychophysical 
studies  of  perceptual  masking  where  short  delays  between  stimuli  result  in 
one  stimuli  overpowering  or  even  inhibiting  the  perception  of  another. 

Turning  next  to  peg  insertion  trials  performed  with  the  aid  of  virtual 
Fixture  6,  v/e  recall  that  virtual  Fixture  6  is  similar  to  virtual  Fixture  4  except 
in  that  the  surface  v/hich  stopped  hand  motion  in  front  of  the  target  hole  is 
removed  and  two  diagonal  surfaces  are  added  in  its  place.  These  two  diagonal 
surfaces  are  positioned  and  oriented  so  as  to  assist  in  the  fine  positioning  of 
the  pog  into  the  target  hole.  When  provided  v/ith  the  real-time  napiic 

information  of  virtual  Fixture  6,  we  again  find  that  the  performance 
degradation  in  the  lime  delayed  iclcnianipuialion  task  was  significanuy  less 

than  in  the  no-fixture  trials.  As  listed  in  Table  11  and  shown  in  Figure  8,  peg 
insertion  trials  using  virtual  Fixture  6  with  time  delays  of  250  ms  and  450  ms 
resulted  in  movement  time  increases  of  4%  and  2%  respectively.  Since 
neither  of  these  results  reflects  statisiicaily  significant  increases  in 
movement  times,  it  is  clear  that  when  using  virtual  Fixture  6,  subjects 
performed  just  as  v^cll  with  250  ms  and  450  ms  time  delays  as  they  did  in  the 
no-  time-delay  trials.  This  result  strongly  confirms  that  virtual  fixtures  can 

be  effective  in  reducing,  and  in  this  case  actually  eliminating,  the 

performance  degradation  due  to  lime  delay. 

In  post  testing  interviews  subjects  were  asked  why  they  fell  their 

performance  was  so  much  belter  with  Fixtures  4  and  6  than  with  no  fixture  in 

time  delay  trials.  Subjects  reported  that  without  the  aid  of  virtual  fixturing 
they  had  to  adopt  a  movc-and-wait  strategy  to  allow  the  feedback  from  the 

remote  site  to  catch  up  with  their  kinesthetic  sense  of  peg  position.  They 
reported  moving  in  front  of  the  target  hole  and  then  waiting  until  the  visual 

feedback  from  the  remote  site  confirmed  that  their  position  was  correct.  With 


the  aid  of  virtual  fixturing,  subjects  were  able  to  use  the  overlaid  real-time 
haptic  cues  in  place  of  the  delayed  visual  cues  and  perform  the  task  without 
having  to  wait  for  visual  confirmation  of  every  action. 

Characteristic  Trajectory  Paths  For  Peg  Insertion  Trials 

In  order  to  confirm  the  reported  peg  insertion  strategies  given  in  post 
testing  interviews  and  gain  further  insight  into  how  virtual  fixtures  enhance 
time  delayed  teleoperation,  subjects  were  asked  to  perform  a  small  number  of 
peg  insertion  trials  with  and  without  the  aid  of  virtual  fixtures  while  their 
hand  trajectories  were  recorded.  Figure  9  through  Figure  11  present 
characteristic  hand  trajectories  for  a  single  subject  performing  a  peg 
insertion  task  for  each  of  the  three  time  delay  configurations.  The  top  half  of 
each  plot  shows  a  bird's  eye  view  of  the  operator's  hand  trajectory  for  a  single 
peg  insertion  trial.  The  plot  depicts  a  motion  in  which  the  peg  starts  at  hole  3, 
the  operator  pulls  away  from  the  task  board,  moves  across  the  board,  and  then 
inserts  the  peg  into  hole  6.  The  horizontal  axis  corresponds  to  motion  across 
the  face  of  the  task  board  and  the  vertical  axis  corresponds  to  distance  away 
from  the  task  board.  The  bottom  half  of  each  plot  shows  the  velocity  profile 
that  corresponds  to  each  of  the  plotted  trajectories.  Although  the  plots  shown 
here  only  represent  a  small  number  of  trials  performed  by  a  single  subject, 
these  particular  plots  were  chosen  for  display  from  the  large  number  of 
recorded  trajectories  because  they  were  typical  and  representative  samples  of 
the  total  data  set. 

Figure  9  shows  sample  trajectories  for  a  subject  performing  a  peg 
insertion  task  from  hole  3  to  hole  6  with  no  time  delay,  250  ms  delay,  and  450 
ms  delay.  For  the  no-delay  trial,  the  position  and  velocity  plots  show  that  the 
subject  pulled  the  peg  out  of  the  board,  moved  it  across  the  board  and  then 
slightly  hesitated  before  inserting  the  peg.  For  the  250  ms  delay  trial  shown 
in  Figure  9,  we  see  a  similar  pattern,  but  the  slight  hesitation  before  peg 
insertion  has  turned  into  a  substantial  pause.  This  pause  before  peg  insertion 
is  reflected  in  the  velocity  profile  as  the  region  where  the  velocity  actually 
drops  to  almost  zero.  This  result  confirms  the  move-and-wait  strategy  that 
subjects  reportedly  adopted  to  allow  the  delayed  visual  feedback  to  confirm 
their  kinesthetic  operations.  For  the  450  ms  delay  trial  shown  in  Figure  9,  the 
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Figure  10.  Sample  Trajectory  Plots  for  FIXTURE  4  Trials  Shown 
for  Each  Time  Delay  Period 
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Figure  11.  Sample  Trajectory  Plots  for  FIXTURE  6  Trials  Shown 
for  Each  Time  Delay  Period 


pause  before  peg  insertion  actually  becomes  a  series  of  pauses  where  the 

subject  moves  and  waits  for  visual  confirmation,  then  adjusts  and  waits  for 
visual  confirmation,  then  adjusts  again  and  waits  for  visual  confirmation.  The 
performance  degradation  in  this  trial  was  significant  compared  to  the  no- 
de!ay  trial.  if  we  compare  the  overall  velocity  plots  of  the  three  time  delay 

scenarios,  it  is  evident  that  velocity  profiles  get  progressively  flatter  as  the 

time  delay  increases.  This  drop  in  overall  velocity  confirms  graphically  the 
performance  degradation  due  to  the  addition  of  time  delay  into  the 

telemanipulation  task. 

Figure  10  shows  trajectory  plots  while  using  Fixture  4  to  perform  the 

peg  insertion  trials  with  the  same  three  time  delay  periods.  A  sketch  of  the 

Fixture  4  surface  profile  is  overlaid  on  top  of  this  graph  to  give  a  rough  idea  of 
where  interaction  with  the  haptic  sensations  occurred  during  the  task. 

Comparing  the  plots  for  the  three  time  delay  periods,  the  hesitation  before  peg 

insertion  is  much  less  pronounced  than  in  the  no-fixture  trials.  This  result 
suggests  that  subjects  were  able  to  use  haptic  sensations  as  confirmation  of 
kinesthetic  operations  and  did  not  have  to  rely  as  heavily  on  visual 

confirmation.  What  is  more,  the  flattening  of  the  velocity  profile  with 

increased  time  delay  is  significantly  less  pronounced  in  these  trials  using 

Fixture  4  than  was  observed  in  the  no-fixture  trials.  This  confirms 

graphically  that  the  performance  degradation  due  to  time  delay  is  less  when 
the  virtual  fixture  was  provided  than  in  the  unaided  trials. 

Figure  11  shows  trajectory  plots  while  using  Fixture  6  to  perform  the 

peg  insertion  trials  with  the  same  three  time  delay  periods.  A  sketch  of  the 

Fixture  6  surface  profile  is  overlaid  on  top  of  this  graph  to  indicate  where 
interaction  with  the  haptic  sensations  occurred  during  the  task.  Comparing 

the  plots  for  the  three  time  delay  periods,  we  find  no  indication  of  hesitation 

before  peg  insertion  for  any  of  the  time  delay  periods.  This  suggests  that, 
using  virtual  Fixture  6,  subjects  were  able  to  use  the  haptic  sensations  in  place 
of  visual  confirmation  to  perform  the  insertion  task.  Not  only  do  the  velocity 
profiles  not  flatten  out  with  increasing  time  delay,  but  the  profiles  actually 

reflect  progressively  faster  trajectories  suggesting  better  performance  in  the 
time  delay  trials  than  in  the  no-delay  trials  when  using  virtual  Fixture  6. 
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These  trajectory  plots  clearly  demonstrate  that,  for  a  given  subject,  the  use  of  a 
virtual  fixture  can  eliminate  the  detrimental  effects  that  time  delay  imparts  on 
telemanipulation. 


CONCLUSIONS 


The  results  of  this  study  confirm  that  overlaying  abstract  sensory 
information  in  the  form  of  virtual  fixtures  on  top  of  the  sensory  feedback  from 
a  remote  environment  can  greatly  reduce  the  drop  in  performance  resulting 
from  time  delays  in  tclemanipulation  tasks.  Without  the  use  of  any  virtual 
fixtures,  36%  and  44%  drops  in  operator  performance  were  recorded  for  time 
delays  of  250  ms  and  450  ms  respectively.  Subjects  reported  adopting  a  move- 
and-wait  strategy  to  deal  with  the  time  delayed  telemanipulation  task  which  was 
confirmed  by  the  recorded  trajectory  plots  of  hand  motion.  The  use  of  virtual 
fixtures  composed  of  simple  combinations  of  rigid  surfaces  were  shown  to 
greatly  reduce  and  in  some  cases  completely  eliminate  the  performance 
degradations  due  to  such  time  delays.  Subjects  reported  being  able  to  use  real¬ 
time  haptic  sensations  as  perceptual  cues  in  place  of  time  delayed  visual  cues. 
As  a  result,  subjects  were  able  to  drop  their  cumbersome  move-and-wait 
technique  and  perform  the  task  more  naturally.  The  absence  of  move-and-wait 
motion  was  confirmed  in  the  recorded  trajectory  plots  of  peg  insertion  trials 
with  virtual  fixtures.  Because  effective  virtual  fixtures  were  developed  from 

basic  haptic  elements  like  rigid  planar  surfaces,  the  development  of  an 
interactive  perceptual  workstation  which  allows  operators  to  build  their  own 
virtual  fixtures  from  basic  building  blocks  seems  like  a  feasible  endeavor  for 
time  delayed  telepresence  tasks.  Such  an  interactive  workstation  could  allow  a 
teleoperator  to  develop  powerful  virtual  fixtures  upon  first  encounter  with  an 
unfamiliar  telemanipulation  task  to  be  performed  in  a  time  delay  situation. 
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